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Highlights  of  GAO-1 1-806,  a  report  to  the 
Committee  on  Armed  Services,  U.S.  Senate 


WEAPONS  ACQUISITION  REFORM 

Actions  Needed  to  Address  Systems  Engineering  and 
Developmental  Testing  Challenges 


Why  GAO  Did  This  Study 

For  the  past  2  years,  the  Department 
of  Defense  (DOD)  has  been 
implementing  the  Weapon  Systems 
Acquisition  Reform  Act  (Reform  Act) 
requirements  for  systems  engineering 
and  developmental  testing.  These 
activities  are  important  to  DOD’s  ability 
to  control  acquisition  costs,  which 
increased  by  $135  billion  over  the  past 
2  years  for  98  major  defense 
acquisition  programs. 

GAO  was  asked  to  determine 

(1)  DOD's  progress  in  implementing 
the  Reform  Act’s  requirements  and 

(2)  whether  there  are  challenges  at  the 
military  service  level  that  could  affect 
their  systems  engineering  and 
developmental  testing  activities.  To  do 
this,  GAO  analyzed  implementation 
status  documents,  discussed 
developmental  testing  office  concerns 
with  current  and  former  DOD  officials, 
and  analyzed  military  service 
workforce  growth  plans  and  test  range 
funding  data. 

What  GAO  Recommends 

GAO  recommends  that  DOD  assess 
the  resources  needed  by  the 
developmental  test  and  evaluation 
office,  develop  a  plan  to  implement  the 
assessment,  develop  metrics  to  aid 
funding  decisions,  and  report  the  effect 
budget  cuts  are  having  on  the  services’ 
ability  to  meet  program  office  needs. 
GAO  also  has  a  matter  for 
congressional  consideration.  DOD 
concurred  with  two  recommendations, 
and  offered  clarifying  language,  which 
GAO  incorporated,  on  the  other  two 
recommendations  for  which  DOD 
partially  concurred. 


View  GAO-1 1-806  or  key  components. 
For  more  information,  contact  Michael  J. 
Sullivan  at  (202)512-4841  or 
sullivanm@gao.gov. 


What  GAO  Found 

The  new  offices  for  systems  engineering  and  developmental  test  and  evaluation 
are  continuing  to  make  progress  implementing  Reform  Act  requirements.  Since 
GAO’s  2010  report  on  this  topic,  the  Deputy  Assistant  Secretaries  for  Systems 
Engineering  and  Developmental  Test  and  Evaluation  have  issued  additional 
policies  and  guidance,  assisted  more  weapons  acquisition  programs  in  the 
development  of  acquisition  plans,  and  provided  input  to  senior  leaders  at 
Defense  Acquisition  Board  meetings.  DOD  also  designated  the  Deputy  Assistant 
Secretary  for  Developmental  Test  and  Evaluation  for  concurrent  service  as  the 
Director  of  the  Test  Resource  Management  Center.  This  was  an  optional  Reform 
Act  provision,  which  places  oversight  of  testing  resources  and  acquisition 
program  developmental  testing  activities  under  one  official.  Despite  these  steps, 
the  developmental  test  and  evaluation  office  reports  having  difficulty  covering  its 
portfolio  of  about  250  defense  acquisition  programs  with  its  current  authorized 
staff  of  63  people.  Current  and  former  testing  officials  believe  the  office  needs 
more  influence  and  resources  to  be  effective,  but  they  said  thorough  analysis  has 
not  been  done  to  determine  the  appropriate  office  size.  Further,  according  to  the 
Deputy  Assistant  Secretary  for  Developmental  Test  and  Evaluation,  a  statutory 
provision  that  designates  the  Test  Resource  Management  Center  as  a  field 
activity  may  limit  his  ability  to  achieve  management  and  reporting  efficiencies  that 
could  be  obtained  by  combining  or  shifting  resources  between  the  two 
organizations.  GAO  has  a  matter  for  Congressional  consideration  to  allow 
shifting  resources  between  the  Test  Resource  Management  Center  and  the 
developmental  test  and  evaluation  office. 

The  military  services  are  facing  workforce  challenges  that  could  curb  systems 
engineering  and  developmental  testing  efforts,  if  not  properly  addressed.  The 
services  planned  to  increase  their  systems  engineering  and  test  and  evaluation 
career  fields  by  about  5,000  people  (14  percent)  and  about  300  people 
(4  percent),  respectively,  between  fiscal  years  2009  and  2015  through  hiring 
actions  and  converting  contractor  positions  to  government  positions.  The 
services  have  increased  the  systems  engineering  career  field  by  about  half  of  its 
projections  and  exceeded  its  planned  growth  for  the  test  and  evaluation  career 
field  through  the  end  of  fiscal  year  2010.  However,  future  growth  may  be  difficult 
because  of  budget  cuts  and  a  clarification  in  DOD’s  insourcing  approach,  which 
may  make  civilian  hiring  more  difficult.  For  example,  the  services  now  plan  to  hire 
about  800  fewer  systems  engineers  by  2015  than  they  originally  projected. 
Further,  cuts  to  development  test  ranges’  fiscal  year  2012  budgets  of  nearly 
$1 .2  billion  (17  percent)  over  the  next  5  years  could  offset  some  of  the  workforce 
gains  already  achieved.  Currently,  the  services  lack  metrics  that  could  be  used  to 
justify  funding  levels,  effectively  allocate  funding  cuts,  make  workforce  decisions, 
or  make  difficult  decisions  related  to  mothballing,  closing,  or  consolidating  test 
capabilities,  if  future  budget  cuts  are  necessary.  To  the  extent  DOD  cannot 
provide  adequate  systems  engineering  and  developmental  testing  support  to  its 
weapon  systems  portfolio,  the  risks  of  executing  the  portfolio  within  cost  and 
schedule  are  increased. 
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MRTFB  Major  Range  and  Test  Facility  Base 
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United  States  Government  Accountability  Office 
Washington,  DC  20548 


September  19,  201 1 

The  Honorable  Carl  Levin 
Chairman 

The  Honorable  John  McCain 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

For  the  past  2  years,  the  Department  of  Defense  (DOD)  has  been 
implementing  provisions  of  the  Weapon  Systems  Acquisition  Reform  Act 
of  20091  (Reform  Act)  related  to  systems  engineering  and  developmental 
testing.  Greater  attention  to  these  activities  provides  the  Office  of  the 
Secretary  of  Defense  an  opportunity  to  affect  weapon  acquisition  plans 
early  and  assess  technical  risks  throughout  weapon  system  development. 
In  addition,  it  could  help  control  total  acquisition  costs,  which  we  recently 
reported2  had  increased  by  $135  billion  over  the  past  2  years  for  DOD’s 
current  portfolio  of  98  major  defense  acquisition  programs.3 

Last  year,  we  reported4  on  the  status  of  DOD’s  initial  efforts  to  implement 
Reform  Act  requirements  related  to  systems  engineering  and 
developmental  testing.  We  found  that  DOD  had  taken  steps  to  implement 
the  Reform  Act  requirements,  including  establishing  new  offices  for  the 
Director  of  Systems  Engineering  and  the  Director  of  Developmental  Test 
and  Evaluation  within  the  Office  of  the  Secretary  of  Defense.  The  offices 
have  since  been  renamed  as  the  offices  of  the  Deputy  Assistant 
Secretary  of  Defense  for  Systems  Engineering  and  the  Deputy  Assistant 
Secretary  of  Defense  for  Developmental  Test  and  Evaluation.  We  found 
that  there  were  concerns  about  the  amount  of  influence  the  Deputy 


Vub.  L.  No.  111-23. 

2GAO,  Defense  Acquisitions:  Assessments  of  Selected  Weapons  Programs, 

GAO-1 1-233SP  (Washington,  D.C.:  Mar.  29,  2011). 

3Research  and  development  costs  grew  over  the  two  year  period  by  an  estimated 
$15  billion,  while  procurement  grew  by  an  estimated  $121  billion.  Of  the  $135  billion  total 
acquisition  cost  increase,  $70  billion  cannot  be  attributed  to  quantity  changes. 

4GAO,  Defense  Acquisitions:  DOD  Needs  to  Develop  Performance  Criteria  to  Gauge 
Impact  of  Reform  Act  Changes  and  Address  Workforce  Issues,  GAO-1 0-774  (Washington, 
D.C.:  July  29,  2010). 
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Assistant  Secretary  for  Developmental  Test  and  Evaluation  could  have  on 
weapon  acquisition  programs  based  upon  where  the  office  is  placed 
organizationally  within  the  department.  However,  we  could  not  determine 
whether  the  office  had  the  appropriate  amount  of  influence  because  it 
was  not  tracking  the  extent  to  which  its  recommendations  were  being 
adopted  or  impacting  weapon  programs.  We  also  identified  issues  the 
military  services  face  as  they  enhance  systems  engineering  and 
developmental  testing  activities  on  their  weapon  acquisition  programs, 
including  determining  whether  they  have  enough  people  to  perform  these 
activities,  training  the  influx  of  new  hires  they  expect,  and  addressing  test 
range  resource  needs. 

Based  on  our  initial  work,  the  Senate  Armed  Services  Committee  asked 
us  to  continue  to  monitor  DOD’s  efforts  to  implement  the  Reform  Act 
provisions,  as  well  as  look  at  the  military  services’  systems  engineering 
and  developmental  test  and  evaluation  capabilities.  Our  specific 
objectives  were  to  determine  (1)  DOD’s  progress  in  implementing  the 
Reform  Act’s  systems  engineering  and  developmental  testing 
requirements  within  the  Office  of  the  Secretary  of  Defense  and 
(2)  whether  there  are  challenges  at  the  military  service  level  that  could 
affect  their  systems  engineering  and  developmental  testing  activities. 

In  conducting  our  work,  we  analyzed  information  obtained  from  the  offices 
of  the  Deputy  Assistant  Secretaries  of  Defense  for  Systems  Engineering 
and  Developmental  Test  and  Evaluation  to  determine  the  status  of  their 
efforts  to  implement  the  Reform  Act  legislation.  We  also  solicited  the 
views  of  current  and  former  DOD  developmental  testing  officials  about  the 
effectiveness  of  the  developmental  test  and  evaluation  office.  In  addition, 
we  interviewed  officials  and  analyzed  pertinent  documents  related  to 
workforce  and  test  range  issues  from  12  DOD  test  ranges,  the  Test 
Resource  Management  Center,  and  cognizant  military  service  systems 
engineering,  developmental  testing,  and  personnel  offices.  For  purposes 
of  this  report,  we  use  the  term  systems  engineering  career  field  to  refer  to 
two  Systems  Planning,  Research  Development,  and  Engineering  career 
fields — Systems  Engineering  and  Program  Systems  Engineering.  The 
test  and  evaluation  career  field  is  a  combination  of  developmental  and 
operational  testing  personnel.  Both  of  these  career  fields  represent  a 
portion  of  the  total  systems  engineering  and  developmental  testing 
workforce.  See  appendix  I  for  a  more  detailed  explanation  of  our  scope 
and  methodology. 

We  conducted  this  performance  audit  from  September  2010  to 
September  2011  in  accordance  with  generally  accepted  government 
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auditing  standards.  Those  standards  require  that  we  plan  and  perform  the 
audit  to  obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable 
basis  for  our  findings  and  conclusions  based  on  our  audit  objectives.  We 
believe  that  the  evidence  obtained  provides  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives. 


Background 


DOD  significantly  downsized  its  acquisition  workforce  in  the  1990s  and 
2000s  as  part  of  an  overall  reduction  of  military  and  civilian  defense 
personnel  after  the  end  of  the  Cold  War.  According  to  DOD’s  April  2010 
Strategic  Human  Capital  Plan  Update ,  the  department  decreased  its  core 
acquisition  workforce5  from  about  146,000  people  in  fiscal  year  1998  to 
about  126,000  in  fiscal  year  2008.  Meanwhile,  the  number  of  major 
defense  acquisition  programs  increased  from  76  to  93  and  total  estimated 
costs6  to  acquire  them  increased  from  nearly  $805  billion  to  more  than 
$1.6  trillion.  The  systems  engineering  and  test  and  evaluation  career 
fields  were  affected  by  the  workforce  cuts.  For  example,  in  2008  the 
Defense  Science  Board  reported7  that  the  Army’s  test  and  evaluation 
workforce  was  reduced  by  more  than  55  percent  from  1991  through  2007. 
In  response  to  cuts  in  these  career  fields,  DOD  reduced  its  emphasis  in 
these  areas  and/or  relied  on  prime  contractors  to  analyze  and  interpret 
developmental  testing  data.  The  Defense  Science  Board  also  found  that 
many  weapon  programs  were  failing  initial  operational  testing  due  to  a 
lack  of  a  disciplined  systems  engineering  approach.  Additionally,  in  2008 
the  National  Academy  of  Sciences  reported8  that  there  were  no  longer 
enough  systems  engineers  to  fill  programs’  needs.  The  Academy 
observed  that  DOD  cannot  outsource  its  technical  and  program 
management  experience  and  intellect  and  still  expect  to  acquire  new 
systems  that  are  both  effective  and  affordable. 


5DOD  defines  the  core  acquisition  workforce  as  all  personnel  whose  responsibilities  are 
predominantly  acquisition  and  assigned  to  positions  designated  as  acquisition. 

6Total  estimated  costs  include  research,  development,  test  and  evaluation,  procurement, 
military  construction,  and  acquisition  operations  and  maintenance  funding. 

7 Report  of  the  Defense  Science  Board  Task  Force  on  Developmental  Test  and  Evaluation 
(Washington,  D.C.:  May  2008). 

8National  Academy  of  Sciences,  Pre-Milestone  A  and  Eariy-Phase  Systems  Engineering: 
A  Retrospective  Review  and  Benefits  for  Air  Force  Future  Systems  Acquisition 
(Washington,  D.C.:  2008). 
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Over  the  past  several  years,  the  Congress  has  called  for  DOD  to  improve 
its  acquisition  workforce  and  increase  emphasis  on  systems  engineering 
and  developmental  testing  during  weapon  systems  development.  For 
example,  the  National  Defense  Authorization  Act  for  Fiscal  Year  20089 
required  the  Secretary  of  Defense  to  establish  the  DOD  Acquisition 
Workforce  Fund  for  the  recruitment,  training,  and  retention  of  DOD 
acquisition  personnel,  which  includes  the  systems  engineering  and  test 
and  evaluation  career  fields.  In  addition,  the  Reform  Act  contains  a 
number  of  systems  engineering  and  developmental  testing  requirements 
aimed  at  helping  weapon  programs  establish  a  solid  foundation  prior  to 
the  start  of  development.  For  example,  the  Deputy  Assistant  Secretaries 
are  expected  to  review  and  approve  acquisition  planning  documents  for 
major  defense  acquisition  programs,  as  well  as  monitor  program 
activities. 

In  response  to  the  Reform  Act  legislation,  DOD  established  new  offices 
within  the  Office  of  the  Secretary  of  Defense  for  systems  engineering  and 
developmental  testing  to  provide  assistance  to  program  offices  as  they 
are  developing  their  acquisition  strategies  prior  to  the  start  of 
development  and  then  oversee  program  office  efforts  to  implement  the 
strategies.  The  systems  engineering  office  has  about  120  programs  in  its 
portfolio  and  the  developmental  test  and  evaluation  office  has  about 
250  programs.  In  addition,  they  provide  advocacy,  oversight  and 
guidance  for  their  respective  workforces  throughout  DOD.  The  Secretary 
of  Defense  also  announced  plans  to  increase  the  number  of  people 
performing  acquisition  activities  by  almost  20,000  employees  through  new 
hiring  actions  and  by  converting  contractor  positions  to  government 
positions  (insourcing)  between  fiscal  years  2009  and  201 5. 10  The 
Strategic  Human  Capital  Plan  Update  indicates  that  22  percent  of  the 
almost  20,000  position  growth  will  be  for  systems  engineering  and  1 
percent  for  test  and  evaluation,  which  would  increase  the  size  of  the 
career  fields  by  16  percent  and  5  percent,  respectively.  The  majority  of 
these  new  positions  will  be  in  the  military  services  at  headquarters, 
program  offices,  and  test  range  locations. 


9Pub.  L.  No.  110-181  §  852. 

10DOD  officials  said  the  decision  to  increase  the  acquisition  workforce  was  in  response  to 
Pub.  L.  No.  110-181  §  852  and  Presidential  Memorandum,  “Government  Contracting,”  74 
Fed.  Reg.  9,755  (Mar.  6,  2009). 
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The  services  maintain  test  ranges  for  development  testing  activities. 
Currently,  24  ranges  are  designated  as  part  of  the  Major  Range  and  Test 
Facility  Base  (MRTFB)11  because  they  have  unique  test  capabilities  that 
are  used  by  multiple  services.  The  Test  Resource  Management  Center  is 
responsible  for  ensuring  that  the  MRTFB  is  adequately  funded  and 
maintained  while  the  services  oversee  the  remaining  non-MRTFB  ranges 
and  facilities,  which  perform  more  service-specific  testing. 


DOD  Is  Continuing  to 
Implement  Reform 
Act  Requirements,  but 
Challenges  Remain 


The  new  offices  of  the  Deputy  Assistant  Secretary  of  Defense  for 
Systems  Engineering  and  Developmental  Test  and  Evaluation  have 
continued  to  make  progress  implementing  the  Reform  Act  requirements. 
Both  offices,  which  provide  assistance  to  program  managers  and  perform 
oversight  for  decision  makers,  have  issued  additional  policies  and 
guidance,  and  are  overseeing  and  providing  information  on  more  weapon 
acquisition  programs  than  they  did  last  year.  The  systems  engineering 
office  will  be  staffed  at  163  people  in  fiscal  year  2012  and  the 
developmental  test  and  evaluation  office  will  be  staffed  at  63  people, 
which  is  less  than  the  developmental  test  and  evaluation  office  originally 
projected.  Yet,  the  offices  are  more  reliant  on  contractor  support  than  the 
Deputy  Assistant  Secretaries  would  like.  In  addition,  many  current  and 
former  testing  officials  continue  to  believe  the  developmental  test  and 
evaluation  office  does  not  have  the  resources  or  influence  to  effectively 
oversee  and  affect  program  decisions.  However,  it  is  unclear  how  many 
people  are  needed. 


Efforts  to  Implement 
Reform  Act  Requirements 
Are  Progressing 


DOD  established  the  two  offices  for  systems  engineering  and 
developmental  test  and  evaluation  within  the  Office  of  the  Secretary  of 
Defense  in  June  2009.  Since  then,  both  offices  have  increased  staffing, 
which  is  enabling  them  to  meet  statutory  requirements  for  assisting  and 
overseeing  their  portfolios  of  defense  acquisition  programs.  The  table 
below  shows  the  actual  and  planned  workforces  for  both  offices. 


^The  Major  Range  and  Test  Facility  Base  is  a  set  of  test  installations,  facilities,  and 
ranges  which  are  regarded  as  “national  assets.”  These  assets  are  sized,  operated,  and 
maintained  primarily  for  DOD  test  and  evaluation  missions.  However,  the  facilities  and 
ranges  are  also  available  to  commercial  and  other  users  on  a  reimbursable  basis. 
Oversight  of  the  facilities  and  ranges  is  performed  by  the  Test  Resource  Management 
Center. 
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Table  1 :  Offices  of  the  Deputy  Assistant  Secretaries  of  Defense  for  Systems 
Engineering  and  Developmental  Test  and  Evaluation  Workforce  Growth 


Workforce  staffing 
(government  and 
contractor  employees) 

June 

2009 

(actual) 

Fiscal  year 
2010 
(actual) 

Fiscal  year 
2011 
(authorized) 

Fiscal  year 
2012 
(planned) 

Goal 

(planned) 

Deputy  Assistant  Secretary  for  Systems  Engineering 

Government 

17 

20 

28 

28 

28 

Contractors 

96 

107 

114 

135 

135 

Total 

113 

127 

142 

163 

163 

Deputy  Assistant  Secretary  for  Developmental  Test  and  Evaluation 

Government 

4 

10 

9 

9 

16 

Contractors 

20 

20 

50 

50 

50 

Total 

24 

30 

63a 

63a 

70a 

Source:  GAO  presentation  of  information  from  the  offices  of  the  Deputy  Assistant  Secretary  of  Defense  for  Systems  Engineering  and 
the  Deputy  Assistant  Secretary  of  Defense  for  Developmental  Test  and  Evaluation. 

alncludes  four  detailees  from  the  Test  Resources  Management  Center. 


Most  of  the  staffing  increases  have  been  through  hiring  contractor 
employees.  While  both  offices  have  also  increased  their  government  staff, 
hiring  freezes  have  curbed  their  ability  to  hire  additional  government 
employees  and  for  the  developmental  test  and  evaluation  office  to  meet 
its  authorized  staffing  goal.  The  systems  engineering  office  was 
authorized  28  government  employees  for  fiscal  year  201 1 ,  but  has  not  yet 
been  given  permission  to  advertise  for  all  of  the  positions  because  of  the 
hiring  freeze.  According  to  the  Deputy  Assistant  Secretary  for 
Developmental  Test  and  Evaluation,  the  office  was  initially  authorized  to 
have  70  employees,  but  will  be  capped  at  63  in  fiscal  year  201 1 , 4  of 
which  are  detailees  from  the  Test  Resource  Management  Center. 

In  their  fiscal  year  2010  joint  annual  report  to  the  Congress,12  the  Deputy 
Assistant  Secretaries  reported  on  their  ongoing  efforts  to  implement 
Reform  Act  requirements.  The  table  below  includes  information  that  was 
included  in  the  report  or  provided  to  us  on  their  fiscal  years  2009  and 
2010  activities.  As  shown,  the  offices  have  continued  or  increased  their 
activities. 


12DOD,  Developmental  Test  and  Evaluation  and  Systems  Engineering  FY  2010  Annual 
Report  (Washington,  D.C.:  Mar.  23,  201 1 ). 
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Table  2:  Ongoing  Efforts  by  Deputy  Assistant  Secretaries  of  Defense  for  Systems  Engineering  and  Developmental  Test  and 
Evaluation  to  Implement  Selected  Reform  Act  Requirements 


Reform  Act  requirement 

Systems  engineering 

Developmental  testing 

Develop  policies  and  guidance 

Fiscal  year  2009:  started  updating  systems 
engineering  plan  guidance  and  the  Guide  for 
Integrating  Systems  Engineering  into  DOD 
Acquisition  Contracts.  Published  policy  that 
expands  reliability,  availability,  and 
maintainability  guidance  for  acquisition 
programs  and  updated  the  Defense 
Acquisition  Guidebook  chapter  on  systems 
engineering. 

Fiscal  year  2010:  published  Directive-Type 
Memoranda  on  (1)  Implementation  of  the 
Weapon  Systems  Acquisition  Reform  Act  of 
2009a  and  (2)  Development  Planning  to 
Inform  Materiel  Development  Decision 
Reviews  and  Support  Analyses  of 
Alternatives.b 

Fiscal  year  2009:  published  guidance  on 
incorporating  test  and  evaluation  requirements 
into  acquisition  contracts.  Updated  required 
content  in  test  and  evaluation  strategy  and 
master  plan  documents  to  include  reliability 
factors. 

Fiscal  year  2010:  published  Information 
Assurance  Policy  Crosswalk  Working  Group 
Report  and  drafted  DOD  Instruction 
establishing  Developmental  Test  and 

Evaluation  responsibilities. 

Review,  approve  acquisition  planning 
documents  of  major  acquisition 
programs 

Fiscal  year  2009:  reviewed  22  and  approved 
16  systems  engineering  plans. 

Fiscal  year  2010:  reviewed  25  and  approved 
21  systems  engineering  plans. 

Fiscal  year  2009:  reviewed  and  approved  25 
developmental  test  and  evaluation  plans. 

Fiscal  year  2010:  reviewed  and  approved  33 
developmental  test  and  evaluation  plans  and  4 
test  and  evaluation  strategies. 

Monitor,  review  activities  of  major 
acquisition  programs 

Fiscal  year  2009:  reviewed  systems 
engineering  activities  for  35  programs  and 
participated  in  20  technical  reviews. 

Fiscal  year  2010:  reviewed  systems 
engineering  activities  for  39  programs  and 
participated  in  31  technical  reviews. 

Fiscal  year  2009:  reviewed  and  assessed 
developmental  testing  activities  for  17 
programs. 

Fiscal  year  2010:  reviewed  and  assessed 
developmental  testing  activities  for  38 
programs. 

Provide  advocacy,  oversight,  and 
guidance  for  respective  DOD 
acquisition  workforce 

Fiscal  year  2009:  started  assessing  systems 
engineering  competencies. 

Fiscal  year  2010:  continued  competency 
assessment  of  workforce.  Sponsored 

Defense  Acquisition  University  course 
development  and  continued  focus  on 
education  and  training. 

Fiscal  years  2009:  updated  education  and 
training  requirements  and  validated 
certification  requirements. 

Fiscal  year  2010:  continued  fiscal  year  2009 
efforts  and  worked  with  components  in 
developing  long-term  strategies  to  optimize 
workforce. 

Source:  GAO  presentation  of  Deputy  Assistant  Secretary  of  Defense  for  Systems  Engineering  and  Deputy  Assistant  Secretary  of 
Defense  for  Developmental  Test  and  Evaluation  information. 

aDOD  Directive-Type  Memorandum  09-027,  Implementation  of  the  Weapon  Systems  Reform  Act  of 
2009  (Dec.  4,  2009,  Incorporating  Change  1,  Oct.  21, 2010). 

bDOD  Directive-Type  Memorandum  10-017,  Development  Planning  to  Inform  Material  Development 
Decision  Reviews  and  Support  Analysis  of  Alternatives  (Sept.  13,  2010,  Incorporating  Change  1, 
May  16,  2011). 


In  addition  to  the  progress  highlighted  in  the  table  above,  the  offices  are 
also  taking  actions  in  areas  that  they  had  not  acted  upon  last  year — 
issuing  required  guidance  on  the  development  and  tracking  of 
performance  criteria  and  exercising  a  Reform  Act  option  of  designating 
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the  Deputy  Assistant  Secretary  of  Defense  for  Developmental  Test  and 
Evaluation  for  concurrent  service  as  the  Director  of  the  Test  Resource 
Management  Center. 

•  Developing  performance  criteria:  The  Reform  Act  requires  the  Deputy 
Assistant  Secretaries,  in  coordination  with  an  official  designated  by 
the  Secretary  of  Defense,  who  was  the  Director  of  the  Performance 
Assessments  and  Root  Cause  Analysis,  to  jointly  issue  guidance  on 
the  development  and  tracking  of  detailed  measurable  performance 
criteria  for  major  defense  acquisition  programs.  In  response  to  this 
requirement,  the  Deputy  Assistant  Secretaries,  in  cooperation  with  the 
Director  of  Performance  Assessments  and  Root  Cause  Analysis,  have 
agreed  that  each  office  shall  develop  guidance  within  their  respective 
functional  areas  in  coordination  with  each  other  and  openly  share  the 
data  and  findings  of  those  performance  criteria  in  conducting  their 
oversight. 


The  Deputy  Assistant  Secretary  for  Systems  Engineering  developed  a 
set  of  time-based  metrics  to  assess  each  program’s  ability  to  execute 
its  system  engineering  plans  and  address  risks  the  office  had 
identified  in  prior  reviews.  The  metrics  measure  program  cost, 
schedule,  staffing,  reliability,  availability  and  maintainability,  software, 
integration,  performance  and  manufacturing,  and  are  to  be 
incorporated  into  each  program’s  systems  engineering  plan  and 
evaluated  at  various  points  in  the  development  process.  Criteria 
developed  by  the  Deputy  Assistant  Secretary  for  Developmental  Test 
and  Evaluation  focus  on  early  acquisition  lifecycle  activities  to  ensure 
that  sound  developmental  testing  planning  is  performed  from  the 
beginning  of  development.  Other  criteria  measure  program  results 
and  are  meant  to  provide  an  objective  foundation  to  assess  a 
program’s  subsequent  developmental  testing  performance  as  it 
approaches  the  production  decision  and  the  assessment  of 
operational  test  readiness.  The  office  plans  to  pilot  test  the  metrics  on 
six  programs,13  with  a  goal  of  rolling  them  out  to  other  programs  by 
the  end  of  201 1 . 


13The  pilot  test  programs  are  the  Integrated  Air  and  Missile  Defense,  MQ-4C  Unmanned 
Aircraft  System  Broad  Area  Maritime  Surveillance,  RQ-4A/B  Unmanned  Aircraft  System 
Global  Hawk,  Joint  Land  Attack  Cruise  Missile  Defense  Elevated  Netted  Sensor  System, 
Patriot  Advanced  Capability-3,  and  the  MQ-9  Unmanned  Aircraft  System  Reaper. 
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•  Changing  leadership  of  Test  Resource  Management  Center:  Effective 
April  1 ,  201 1 ,  the  Principal  Deputy  Under  Secretary  of  Defense  for 
Acguisition,  Technology  and  Logistics  designated  the  Deputy 
Assistant  Secretary  of  Defense  for  Developmental  Test  and 
Evaluation  for  concurrent  service  as  the  Director  of  the  Test  Resource 
Management  Center.  DOD  had  not  acted  upon  this  optional  Reform 
Act  provision  last  year.  Both  offices  will  continue  to  be  managed 
separately  and  report  to  different  authorities — developmental  test  and 
evaluation  office  activities  will  be  reported  to  the  Assistant  Secretary 
of  Defense  for  Research  and  Engineering  and  Test  Resource 
Management  Center  activities  will  be  reported  directly  to  the  Under 
Secretary  of  Defense  for  Acguisition,  Technology  and  Logistics.14 

According  to  the  Deputy  Assistant  Secretary  of  Defense  for 
Developmental  Test  and  Evaluation,  the  Principal  Deputy  Under 
Secretary  for  Acguisition,  Technology  and  Logistics  asked  him  to 
complete  a  study  in  the  near  future  to  identify  efficiencies  that  can  be 
obtained  by  merging  some  of  the  offices’  activities.  A  study  was 
started  in  July  201 1 .  The  Deputy  Assistant  Secretary  for 
Developmental  Test  and  Evaluation  indicated  that  there  could  be 
some  limitations  on  his  ability  to  streamline  management  and 
reporting  activities  or  shift  resources  between  the  organizations 
because  the  Test  Resource  Management  Center  is  designated  by 
statute  to  be  a  field  activity  and  the  organizations  are  funded 
separately.  DOD  has  not  studied  the  possible  legal  ramifications  of 
combining  the  offices. 

In  their  fiscal  year  201 0  joint  annual  report  to  the  Congress,  the  Deputy 
Assistant  Secretaries  also  identified  several  focus  areas  for  improvement. 
For  example,  the  systems  engineering  office  plans  to  reestablish  the  DOD 
Software  Working  Group  to  improve  DOD’s  capability  to  address  systemic 
software  program  issues.  Among  other  things,  the  developmental  test  and 


14The  Bob  Stump  National  Defense  Authorization  Act  for  Fiscal  Year  2003  provided  for 
the  establishment  of  the  Test  Resource  Management  Center  as  a  field  activity  that  would 
report  directly  to  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics.  The  Reform  Act  established  the  Director  of  Developmental  Test  and  Evaluation 
within  the  Office  of  the  Secretary  of  Defense,  but  did  not  dictate  how  that  official’s  office 
should  be  aligned.  The  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  decided  to  place  the  office  under  the  Director  of  Defense,  Research  and 
Engineering,  which  has  since  been  renamed  the  Assistant  Secretary  of  Defense  for 
Research  and  Engineering. 
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evaluation  office  wants  to  develop  a  responsible  test  organization  model 
that  the  services  would  use  to  designate  the  lead  government  test 
organization  responsible  for  overseeing  and/or  conducting  the 
developmental  test  and  evaluation  for  an  acquisition  program.  In  addition, 
the  office  wants  to  issue  a  policy  requiring  programs  to  prioritize  use  of 
government  capabilities  and  to  provide  a  cost-benefit  analysis  when  they 
decide  to  provide  funding  to  use  and/or  develop  test  capabilities  at  prime 
contractor  sites. 

In  a  departure  from  the  fiscal  year  2009  joint  annual  report,15  the  fiscal 
year  201 0  report  did  not  contain  a  discussion  of  the  extent  to  which 
weapon  acquisition  programs  are  fulfilling  their  systems  engineering  or 
test  and  evaluation  master  plans.16  Further,  the  fiscal  year  2010  report  did 
not  include  a  discussion  of  test  and  evaluation  waivers  or  deviations  that 
programs  have  received.  Instead,  the  report  identified  the  type  of  reviews 
or  engagements  both  of  the  offices  participated  in  for  various  programs. 
The  Deputy  Assistant  Secretaries  stated  that  they  did  not  provide  the 
information  in  the  fiscal  year  2010  report  because  they  were  directed  to 
streamline  the  report.  In  May  201 1 ,  the  Senate  Armed  Services 
Committee  requested  DOD  to  supplement  the  fiscal  year  2010  report  with 
this  information  as  required  by  the  Reform  Act.  DOD  has  not  yet  provided 
the  information. 

The  Deputy  Assistant  Secretaries  believe  they  have  had  a  positive 
influence  on  weapon  acquisition  programs  over  the  past  2  years  during 
milestone  reviews  with  senior  department  leaders  and  through 
recommendations  to  program  offices.  Although  the  Assistant  Secretary  of 
Defense  for  Research  and  Engineering  represents  both  systems 
engineering  and  developmental  test  positions  at  Defense  Acquisition 
Board  meetings,  the  Deputy  Assistant  Secretaries  said  that  either  they,  or 


15DOD,  Developmental  Test  and  Evaluation  and  Systems  Engineering  FY  2009  Annual 
Report  (Washington,  D.C.:  Mar.  2010). 

16The  Reform  Act  requires  the  joint  annual  report  to  Congress  to  include  (1 )  a  discussion 
of  the  extent  to  which  the  major  defense  acquisition  programs  are  fulfilling  the  objectives 
of  their  systems  engineering  master  plans  and  developmental  test  and  evaluation  plans; 

(2)  a  discussion  of  the  waivers  of  and  deviations  from  requirements  in  test  and  evaluation 
master  plans,  systems  engineering  plans  and  other  testing  requirements,  any  concerns 
raised  by  such  waivers  or  deviations,  and  the  actions  taken  to  address  such  concerns; 
and  (3)  an  assessment  of  the  organization  and  capabilities  of  DOD  for  systems 
engineering,  development  planning,  and  developmental  test  and  evaluation  with  respect  to 
such  programs. 


Page  10 


GAO-1 1-806  Weapons  Acquisition  Reform 


designees,  also  attend  the  meetings.  They  said  that  they  have  been 
asked  by  the  Principal  Deputy  for  Acquisition,  Technology  and  Logistics 
to  provide  direct  input  about  weapon  acquisition  programs  at  some  of 
these  meetings,  including  discussions  on  the  Gray  Eagle  unmanned 
aircraft  system,  Ohio  Replacement  submarine,  and  the  Joint  Strike 
Fighter  aircraft  programs.  The  Deputy  Assistant  Secretaries  also  said  that 
program  offices  are  making  changes  based  on  the  recommendations 
made  by  their  offices  during  regular  program  assessments  and  technical 
reviews. 


Organizational  Challenges  While  the  Deputy  Assistant  Secretaries  have  made  progress 
Remain  implementing  the  Reform  Act  requirements,  we  identified  several 

organizational  challenges  that  could  limit  their  effectiveness.  A  summary 
of  each  of  these  challenges  is  presented  below. 

•  Reliance  on  contractor  employees:  The  Deputy  Assistant  Secretaries 
rely  heavily  on  contractors  to  help  perform  office  activities.  In  fiscal 
year  2010,  for  example,  nearly  85  percent  of  the  staff  in  the  systems 
engineering  office  and  67  percent  of  the  staff  in  the  developmental 
test  and  evaluation  office  were  contractors.  Both  Deputy  Assistant 
Secretaries  would  like  to  have  a  larger  proportion  of  government 
employees  because  they  believe  it  is  important  to  maintain  a  core 
cadre  of  people  with  the  required  institutional  knowledge  and  skills  to 
support  current  and  future  program  office  needs.  However,  they  are 
not  optimistic  about  their  chances  of  getting  additional  government 
employees  because  of  a  civilian  hiring  freeze. 

•  Developmental  Test  and  Evaluation  Office  influence:  Current  and 
former  DOD  test  and  evaluation  officials  continue  to  believe  the 
developmental  test  and  evaluation  office  could  be  more  effective  in  its 
oversight  role  with  the  proper  influence.  For  example,  they  pointed  out 
that  the  office’s  primary  avenue  for  voicing  concerns  about  weapon 
acquisition  programs  to  senior  leaders  is  at  overarching  integrated 
product  team  meetings  that  take  place  in  preparation  for  Defense 
Acquisition  Board  meetings.  The  integrated  product  team  leader17 


17Depending  on  the  acquisition  program,  the  integrated  product  team  leader  could  be  the 
Deputy  Assistant  Secretary  of  Defense  for  Strategic  and  Tactical  Systems,  the  Deputy 
Assistant  Secretary  of  Defense  for  Space  and  Intelligence,  the  Deputy  Assistant  Secretary 
of  Defense  for  C3,  Space  and  Spectrum,  or  the  Overarching  Integrated  Product  Team 
Leader  for  Defense  Business  Systems  (Office  of  the  Deputy  Chief  Management  Officer). 


Page  11 


GAO-1 1-806  Weapons  Acquisition  Reform 


ultimately  decides  which  organizations  will  get  to  present  issues  at  the 
Defense  Acquisition  Board  meetings.  Current  testing  officials  told  us 
that  in  some  cases  developmental  testing  issues  do  not  make  the 
Defense  Acquisition  Board  meeting  agendas,  which  is  a  concern  to 
officials  who  believe  the  developmental  test  and  evaluation  office 
should  provide  independent  assessments  of  weapon  acquisition 
programs  directly  to  senior  leaders.  In  addition,  officials  said  that  the 
deputy  directors  who  attend  the  overarching  meetings  are  not  at  the 
senior  executive  level  like  other  meeting  attendees,  which  officials 
said  in  some  cases  has  reduced  their  relative  influence  during  these 
meetings.  To  ensure  developmental  testing  information,  such  as  the 
assessment  of  operational  test  readiness,  receives  appropriate 
consideration  by  senior  leaders,  the  Defense  Science  Board 
recommended  in  2008  that  the  office  report  developmental  testing 
issues  directly  to  the  Deputy  Under  Secretary  of  Defense  for 
Acquisition  and  Technology.  Currently,  the  office  reports  through  an 
intermediary — the  Assistant  Secretary  of  Defense  for  Research  and 
Engineering. 


Even  though  the  systems  engineering  office  reports  through  the  same 
reporting  channel  as  the  developmental  test  and  evaluation  office,  the 
Deputy  Assistant  Secretary  for  Systems  Engineering  believes  his 
office  has  the  appropriate  amount  of  influence.  This  is  because  the 
systems  engineering  office’s  primary  emphasis  is  on  assisting 
program  managers  in  the  development  of  their  systems  engineering 
plans.  In  contrast,  the  Deputy  Assistant  Secretary  for  Developmental 
Test  and  Evaluation  believes  his  office  should  put  about  equal  effort 
into  assisting  and  assessing  program  office  activities. 

•  Developmental  Test  and  Evaluation  Office  resources:  Information 
provided  by  the  developmental  test  and  evaluation  office  shows  the 
office  can  not  provide  full  coverage  of  its  portfolio  of  about  250 
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acquisition  programs  given  its  current  workforce.18  For  example,  in 
fiscal  year  2010  when  the  office  had  30  people,  89  programs 
(36  percent)  did  not  receive  any  support,  including  the  Air  and  Missile 
Radar  program  and  sub  elements  of  the  Early  Infantry  Brigade 
Combat  Team  program,  such  as  the  Small  Unmanned  Ground 
Vehicles  and  Unmanned  Aircraft  System. 

Although  staffing  has  doubled  between  fiscal  years  2010  and  2011, 
the  office  still  has  not  been  able  to  support  all  the  programs  the 
Deputy  Assistant  Secretary  for  Developmental  Test  and  Evaluation 
believes  it  should.  The  Deputy  Assistant  Secretary  for  Developmental 
Test  and  Evaluation  said  the  office  has  had  to  be  selective  in  using  its 
resources  and  he  has  introduced  a  “triage”  strategy  for  dealing  with 
the  overload  of  programs  relative  to  the  office’s  workforce.  This 
strategy  includes  dropping  virtually  all  programs  below  Acquisition 
Category  I  from  the  developmental  testing  oversight  list  and 
eliminating  oversight  of  some  major  automated  information  systems. 
For  the  most  part,  the  office  focuses  its  efforts  on  major  defense 
acquisition  programs  between  milestone  B  (development  start)  and 
milestone  C  (the  production  decision)  in  order  to  retain  sufficient  depth 
with  programs  in  development.  It  is  providing  minimal  coverage  to 
programs  prior  to  the  start  of  development,  which  is  the  most 
opportune  time  to  influence  a  program’s  acquisition  strategy.  Based 
on  the  “triage”  strategy,  the  Deputy  Assistant  Secretary  for 
Developmental  Test  and  Evaluation  believes  only  about  half  of  the 
current  portfolio  of  about  250  programs  would  receive  the  level  of 
support  he  believes  is  needed  with  a  staff  of  63  people.  On  the  other 
hand,  the  Deputy  Assistant  Secretary  for  Systems  Engineering 


1 ft 

The  portfolio  includes  a  combination  of  Acquisition  Category  ID,  Acquisition  Category 
1C,  Major  Automated  Information  Systems,  and  other  programs  that  are  specified  as 
special  interest  by  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  or  on  the  Deputy  Assistant  Secretaries’  oversight  list.  Acquisition  Category  I 
programs  are  major  defense  acquisition  programs  estimated  by  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and  Logistics  to  require  eventual  expenditure  for 
research,  development,  test  and  evaluation  of  more  than  $365  million  in  fiscal  year  2000 
constant  dollars  or,  for  procurement,  of  more  than  $2.19  billion  in  fiscal  year  2000  constant 
dollars,  or  those  designated  by  the  Milestone  Decision  Authority  to  be  special  interest 
programs.  For  Acquisition  Category  ID  programs,  the  milestone  decision  authority  is  the 
Under  Secretary  for  Acquisition,  Technology  and  Logistics.  The  “D”  refers  to  the  Defense 
Acquisition  Board,  which  advises  the  Under  Secretary  at  major  decision  points.  For 
Acquisition  Category  1C  programs,  the  milestone  decision  authority  is  the  DOD 
component  head  or,  if  delegated,  the  DOD  component  acquisition  executive.  The  “C” 
refers  to  component. 
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believes  his  office,  which  is  supposed  to  have  163  people  in  fiscal 
year  2012,  will  have  enough  staff  to  oversee  its  portfolio  of  120 
programs. 

While  current  and  former  DOD  testing  officials  provided  reasons  for 
increasing  the  size  of  the  developmental  test  and  evaluation  office, 
they  could  not  specify  the  appropriate  size  for  the  office,  as  they 
indicated  the  issue  has  not  been  thoroughly  analyzed.  Officials 
familiar  with  the  establishment  of  the  office  told  us  that  three  staffing 
scenarios  were  considered  prior  to  the  office  being  established — a 
high,  medium,  and  low  staffing  scenario — but  they  indicated  that  no 
detailed  analysis  was  done  to  support  any  of  the  scenarios.  Under  the 
high  staffing  scenario,  the  developmental  test  and  evaluation  office 
would  have  had  250  people,  which  would  have  matched  the  size  of 
the  office  of  the  Director  of  Operational  Test  and  Evaluation.  The 
medium  level  of  staffing,  120-150  people,  was  based  on  the  size  of  a 
legacy  developmental  testing  organization  and  the  low  level,  which 
called  for  90  people,  was  based  on  an  assumption  about  the  number 
of  programs  each  person  would  be  responsible  for  overseeing.  As 
shown  earlier  in  table  1,  the  staffing  goal  is  70  people,  which  is  fewer 
than  the  lowest  staffing  scenario  considered. 

Former  testing  officials  believe  an  opportunity  exists  to  both  increase 
the  developmental  testing  office’s  influence  and  address  resource 
concerns  by  merging  Test  Resource  Management  Center  and 
developmental  testing  office  activities.  They  believe  this  would  give 
the  Deputy  Assistant  Secretary  for  Developmental  Test  and 
Evaluation  the  most  flexibility  in  how  to  allocate  resources.  They  also 
pointed  out  that  in  the  early  1990s,  oversight  of  all  developmental  test 
and  evaluation  activities  had  been  under  one  organization  that 
reported  directly  to  the  Under  Secretary  of  Defense  for  Acquisition.19 


1 Q 

The  office  of  the  Under  Secretary  of  Defense  for  Acquisition  is  now  the  office  of  the 
Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics. 
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Workforce  Challenges 
Could  Affect  Services’ 
Systems  Engineering 
and  Developmental 
Testing  Efforts 


The  military  services  have  been  increasing  the  number  of  people  in  their 
systems  engineering  and  test  and  evaluation  career  fields,  but  challenges 
exist  that  could  impede  future  workforce  growth  plans  as  well  as  testing  at 
the  ranges,  if  not  properly  addressed.  The  services  planned  to  increase 
their  systems  engineering  and  test  and  evaluation  career  fields  by  about 
5,000  people  (14  percent)  and  about  300  people  (4  percent),  respectively, 
between  fiscal  years  2009  and  2015  through  hiring  actions  and  by 
insourcing  contractor  positions.  These  increases  are  part  of  DOD’s  overall 
efforts  to  increase  the  number  of  people  in  acquisition  career  fields.  The 
services  have  increased  the  systems  engineering  career  field  by  about 
half  of  their  projections  and  exceeded  their  planned  growth  for  the  test 
and  evaluation  career  field  through  the  end  of  fiscal  year  2010.  However, 
budget  cuts  and  a  clarification  in  DOD’s  insourcing  approach  may  make 
hiring  civilians  more  challenging  in  the  future.  The  services’ 
developmental  test  ranges  are  also  experiencing  declining  budgets,  as 
the  fiscal  year  2012  budget  includes  cuts  of  nearly  $1 .2  billion  over  the 
next  5  years  to  support  accounts  that  pay  for  overhead  costs.  The 
services  plan  to  cut  range  personnel  in  response  to  the  budget 
reductions,  which  could  offset  some  of  the  workforce  gains  they  have 
already  achieved.  They  have  not  yet  determined  how  the  cuts  will  be 
allocated  across  the  ranges  or  the  impact  they  will  have  on  meeting 
program  office  needs.  Further  budget  cuts  are  possible  based  on  the 
recent  debt  ceiling  agreement,  but  details  are  unknown.  Currently,  the 
services  lack  common  performance  metrics  that  would  assist  in  making 
funding  decisions. 


Services  Have  Increased 
Their  Systems  Engineering 
and  Test  and  Evaluation 
Career  Fields 


The  services  planned  to  increase  their  acquisition  systems  engineering 
and  test  and  evaluation  career  fields  by  14  percent  and  4  percent, 
respectively,  between  fiscal  years  2009  and  2015  through  hiring  and 
insourcing  actions.  This  would  increase  the  overall  systems  engineering 
career  field  by  about  5,000  people  from  about  35,000  to  40,000  people 
and  the  test  and  evaluation  career  field  by  almost  300  people  from  about 
7,400  to  7,700  people.  These  new  positions  would  be  located  at  service 
headquarters,  program  offices,  and  test  range  locations.  It  should  be 
noted  that  insourcing  actions  alone  do  not  result  in  real  growth  to  the 
collective  number  of  civilians,  military,  and  contractors  performing  an 
activity  because  it  only  involves  the  transfer  of  positions  from  contractors 
to  the  government.  Hiring  additional  civilian  employees,  on  the  other 
hand,  would  result  in  growth,  assuming  the  contractor  and  military 
workforces  remain  stable. 
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The  following  table  shows  the  baseline,  goal,  and  current  number  of 
civilian  and  military  personnel  performing  systems  engineering  and  test 
and  evaluation  activities  for  each  of  the  services  at  the  end  of  fiscal  year 
2010,  as  well  as  the  percentage  of  the  growth  goal  target  achieved. 


Table  3:  Actual  and  Planned  Military  Service  Systems  Engineering  and  Test  and  Evaluation  Career  Field  Growth 


Baseline3 

Growth  goal  by  the  end 
of  fiscal  year  201 5b 

Current  workforce  as 
of  September  30,  2010 

Percentage  of  growth  goal 
target  achieved  as  of 
September  30,  2010 

Systems  engineering  career  field 

Air  Force 

6,380 

7,059 

7,575 

107% 

Army 

10,615 

12,076 

10,938 

91% 

Navy 

17,961 

20,870 

19,012 

91% 

Totals 

34,956 

40,005 

37,525 

94% 

Test  and  evaluation  career  field 

Air  Force 

2,622 

2,566 

2,840 

111% 

Army 

2,135 

2,297 

2,211 

96% 

Navy 

2,652 

2,829 

2,977 

105% 

Totals 

7,409 

7,692 

8,028 

104% 

Source:  Military  service  workforce  numbers  presented  in  DOD’s  April  2010  Strategic  Human  Capital  Plan  Update  and  military  service 
budget  estimate  submissions. 

Note:  Workforce  numbers  include  civilian  and  military  positions.  Navy  systems  engineering  workforce 
also  includes  science  and  technology  positions,  which  officials  estimate  to  be  less  than  2  percent  of 
the  total. 

aAir  Force  and  Army  baseline  is  the  end  of  fiscal  year  2008  and  the  Navy  baseline  is  the  end  of  fiscal 
year  2009. 

bAir  Force  and  Army  projections  are  based  on  the  fiscal  year  2010  budget  estimate  submission  and 
the  Navy  projections  are  based  on  the  fiscal  year  201 1  budget  estimate  submission. 


Through  the  end  of  fiscal  year  2010,  the  services  have  made  significant 
progress  towards  increasing  the  two  career  fields.  For  example,  the 
services  increased  the  systems  engineering  career  field  by  about 
2,600  people,  about  half  of  the  projected  growth.  Collectively,  they  have 
achieved  94  percent  of  the  growth  goal  target  planned  for  that  career 
field.  The  services  achieved  104  percent  of  the  growth  goal  target 
planned  for  the  test  and  evaluation  career  field  by  adding  over 
600  people.  Information  provided  by  the  Army  and  Navy  shows  about 
74  percent  of  their  systems  engineering  career  field  growth  was  through 
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new  hires  and  the  remaining  26  percent  was  through  insourcing.20  About 
86  percent  of  their  test  and  evaluation  career  field  growth  was  through 
new  hires  and  the  remaining  14  percent  was  through  insourcing. 


Budget  Cuts  and  Other  Recently  proposed  budget  changes  would  result  in  modifications  to  the 

Challenges  Could  Affect  services’  workforce  growth  plans  for  both  acquisition  career  fields.21  The 

Overall  Workforce  Growth  services  still  plan  to  increase  the  systems  engineering  career  field 

between  fiscal  years  201 1  and  2015,  but  with  about  800  positions  less 
than  originally  planned.  As  a  result,  career  field  growth  would  be 
10  percent  instead  of  14  percent.  On  the  other  hand,  the  services  plan  to 
hire  400  more  people  than  they  originally  planned  for  the  test  and 
evaluation  career  field,  despite  the  fact  that  the  overall  number  of  DOD 
civilians  is  frozen  at  fiscal  year  2010  levels.  This  would  result  in  a 
6  percent  growth  to  the  career  field  instead  of  4  percent. 

Service  officials  stated  that  achieving  additional  career  field  growth, 
however,  could  be  difficult  because  of  a  recent  clarification  in  DOD’s 
insourcing  policy.  According  to  a  March  201 1  memorandum  issued  jointly 
by  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  and  the  Under  Secretary  of  Defense  Comptroller/Chief  Financial 
Officer,  a  case-by-case  approach  will  be  used  for  additional  insourcing  of 
acquisition  functions  based  on  critical  need,  whether  a  function  is 
inherently  governmental,  and  the  benefit  demonstrated  by  a  cost-benefit 
analysis.  The  memorandum  also  states  that  additional  insourcing  must  be 
supported  by  current  budget  levels.  In  cases  where  added  insourcing 
would  breach  the  existing  civilian  ceilings,  the  proposal  and  associated 
justification  must  be  provided  to  the  Director  of  Human  Capital  Initiatives 
and  then  the  proposal  will  be  reviewed  by  the  two  Under  Secretaries  of 
Defense  issuing  the  memorandum  and  approved  by  the  Deputy  Secretary 
of  Defense.  Of  the  nearly  2,000  people  the  Army  and  Navy  now  plan  to 
hire  for  the  systems  engineering  career  field  between  fiscal  years  201 1 
and  2015,  96  percent  is  to  be  achieved  through  insourcing.  And  about 


The  Air  Force  could  not  provide  a  breakout  of  the  number  of  positions  it  plans  to 
insource  between  fiscal  years  2010  and  2015.  The  Air  Force  only  maintains  information  on 
its  broad  plans  to  insource,  as  they  allow  their  major  commands  to  determine  the  specific 
positions  that  are  insourced. 

21  Under  the  Budget  Control  Act  of  2011,  budgetary  resources  may  be  further  reduced,  but 
details  about  the  effect,  if  any,  on  workforce  growth  plans  are  unknown  at  this  time.  Pub. 

L.  No.  112-25,  §  101. 
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40  percent  of  the  over  200  people  they  plan  to  hire  for  the  test  and 
evaluation  career  field  are  to  be  through  insourcing. 


Although  the  services  have  increased  their  test  and  evaluation  staff,  each 
of  the  12  test  ranges  we  visited  experienced  a  mixture  of  recruiting,  hiring, 
training,  or  retention  challenges.  For  example,  Pacific  Missile  Range 
Facility  officials  stated  the  range’s  location  on  the  island  of  Kauai,  Hawaii, 
has  made  it  difficult  to  recruit  personnel  due  to  a  lower  pay  grade 
structure  and  higher  costs  of  living  compared  to  other  test  ranges.  Range 
officials  stated  that  it  can  take  months  to  hire  new  employees,  forcing 
many  qualified  applicants  to  seek  employment  opportunities  elsewhere. 
Test  ranges  have  begun  using  acquisition  workforce  expedited  hiring 
authority,  which  allows  DOD  components  to  streamline  the  process  for 
making  offers  to  qualified  acquisition  personnel.  However,  ranges  have 
interpreted  this  authority  differently.  Service  test  and  evaluation 
executives  said  they  would  clarify  the  policy  to  the  ranges  based  upon  our 
observations  to  ensure  the  policies  are  fully  understood.  Most  of  the  test 
range  officials  we  spoke  with  also  had  concerns  about  the  timeliness  and 
quality  of  the  Defense  Acquisition  University’s  training  classes.22  The 
Defense  Acquisition  University  is  coordinating  with  the  services  to 
address  the  increased  demand  for  acquisition  training.  Finally,  some  test 
ranges  are  having  difficulty  retaining  engineers  or  some  software 
specialties  because  the  private  sector  can  pay  them  a  higher  salary. 
Officials  at  Aberdeen  Test  Center,  Maryland,  are  concerned  that  many  of 
their  employees  will  take  higher  paying  jobs  with  organizations  that  are 
moving  into  their  area  as  the  result  of  a  Base  Realignment  and  Closure 
decision. 


22According  to  DOD,  each  acquisition,  technology,  and  logistics  position,  meaning 
positions  designated  to  be  acquisition  positions  based  on  the  Defense  Acquisition 
Workforce  Improvement  Act,  has  certification-level  requirements.  When  an  individual  is 
placed  in  one  of  these  positions,  the  DOD  component  is  required  to  either  document  the 
determination  that  the  individual  has  satisfied  appropriate  certification  and  assignment- 
specific  training  requirements,  or  establish  a  plan  for  the  individual  to  meet  the 
requirements  within  24  months  of  placement  or  other  established  period.  If  an  individual 
does  not  meet  position  requirements  within  established  time  frames,  a  waiver  must  be 
obtained  according  to  applicable  procedures  for  the  individual  to  remain  in  the  position. 
Department  of  Defense  Instruction  5000.66,  Operation  of  the  Defense  Acquisition , 
Technology ,  and  Logistics  Workforce  Education,  Training,  and  Career  Development 
Program  at  E2.4.1 .2.  (Dec.  21 , 2005). 
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Impact  of  Proposed 
Budget  Cuts  on  Ranges 
Has  Not  Been  Determined 


The  services’  developmental  test  range  budgets  are  being  reduced,  but 
the  full  impact  of  the  cuts  has  yet  to  be  determined.  The  fiscal  year  2012 
President’s  Budget  includes  cuts  of  nearly  $1 .2  billion  (17  percent) 
through  fiscal  year  2015  to  the  ranges’  institutional  funding  accounts  that 
fund  operational  and  overhead  expenses  such  as  personnel,  facilities, 
and  equipment  costs.23  According  to  service  officials,  the  budget  cuts  are 
a  result  of  direction  they  received  late  in  the  fiscal  year  2012  budget  cycle 
to  keep  their  civilian  workforce  at  the  same  level  as  they  had  at  the  end  of 
fiscal  year  2010.  Research,  development,  test  and  evaluation  accounts 
that  fund  range  testing  activities  absorbed  a  large  portion  of  the  cuts. 

As  shown  in  figure  1,  the  services  had  already  planned  to  reduce  their 
MRTFB  research,  development,  test  and  evaluation  funding  between 
fiscal  years  2012  and  2015.  Although  range  budgets  reached  a  peak  in 
fiscal  year  201 1 ,  budget  estimates  for  that  year  projected  funding 
decreases  in  subsequent  years.  According  to  service  officials  these 
reductions  were  expected  due  to  expected  workload  reductions,  savings 
associated  with  implementing  efficiency  initiatives,  and  a  civilian  pay 
freeze.  The  fiscal  year  2012  budget  contains  additional,  more  significant 
decreases  beyond  those  forecasted  in  the  previous  year’s  budget.  In 
total,  this  budget  provides  $5.7  billion  for  the  ranges  between  fiscal  years 
2012  and  2015 — a  17  percent  reduction  from  the  $6.9  billion  forecast  in 
the  fiscal  year  201 1  budget.  Most  notably,  the  Army  and  Air  Force  range 
budgets  were  each  cut  over  $100  million  in  fiscal  year  2012  alone. 


no 

Under  the  Budget  Control  Act  of  201 1 ,  budgetary  resources  may  be  further  reduced,  but 
details  about  the  effect,  if  any,  on  workforce  growth  plans  are  unknown  at  this  time.  Pub. 

L.  No.  112-25,  §  101. 
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Figure  1:  MRTFB  Operation  &  Support  Funding  for  Fiscal  Years  2006-2015 
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Source:  GAO  presentation  of  military  service  Information  from  the  fiscal  year  2011  and  fiscal  year  2012  President's  Budget 


The  Chairman  of  the  Senate  Armed  Services  Committee  recently 
expressed  concern  that  these  proposed  reductions  threaten  to  seriously 
undermine  the  implementation  of  the  Reform  Act’s  requirement  to  rebuild 
DOD’s  systems  engineering  and  developmental  testing  organizations  and 
requested  that  the  proposed  cuts  be  reviewed.  The  Army  and  Air  Force 
are  in  the  process  of  reviewing  these  cuts  and  determining  where  they  will 
be  made.  To  minimize  the  impact  on  these  activities,  they  are  examining 
potential  reductions  to  their  overhead  or  administrative  staff  at 
headquarters  and  field  locations.  According  to  service  headquarters 
officials,  if  DOD’s  proposed  budget  cuts  between  fiscal  years  2012  and 
2015  remain  intact,  they  may  have  to  cut  some  of  their  developmental 
test  capabilities  as  well  as  personnel  who  conduct  tests.  This  could  offset 
some  of  the  test  and  evaluation  career  field  gains  already  achieved  over 
the  past  2  years.  Test  officials  said  that  reductions  in  test  personnel  could 
limit  the  amount  of  testing  performed  on  weapon  acquisition  programs, 
which  could  increase  the  risk  associated  with  those  programs  and/or 
result  in  an  extension  of  a  program’s  test  schedule.  Final  decisions  on 
where  to  take  the  cuts  have  not  yet  been  made.  Therefore,  we  could  not 
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determine  the  impact  funding  cuts  would  have  on  the  ranges’  ability  to 
meet  program  office  testing  needs. 

Range  officials  indicated  that  prior  to  these  proposed  cuts  they  had 
difficulty  maintaining  their  test  capabilities.  The  National  Defense 
Authorization  Act  for  Fiscal  Year  200324  required  the  Secretary  of 
Defense  by  fiscal  year  2006  to  establish  the  funding  objective  of  ensuring 
that  the  overhead  and  institutional  costs  of  the  MRTFB  ranges  were  fully 
funded  through  the  major  test  and  evaluation  investment  accounts  and  to 
ensure  that  DOD  customers  were  charged  not  more  than  the  direct  costs 
of  testing.  The  law  also  required  DOD  to  establish  the  Test  Resource 
Management  Center  and  required  that  the  director  of  that  organization 
certify  whether  the  services’  proposed  budgets  for  test  and  evaluation 
activities  are  adequate.  To  comply  with  this  law,  the  services  increased 
their  range  operating  budgets  by  over  50  percent  in  fiscal  year  2006. 
Although  MRTFB  funding  was  fairly  stable  between  fiscal  years  2006  and 
201 1,25  range  officials  said  they  have  had  difficulties  maintaining  their  test 
capabilities  because  the  infrastructure  is  aging  and  operating  costs  for 
expenses  like  utilities  and  fuel  have  grown  at  a  higher  rate  than  their 
overall  funding.  As  a  result,  officials  said  they  have  had  to  move  money 
from  their  range  modernization  accounts  to  fund  operating  costs,  only  fix 
things  that  are  broken  or  in  emergency  status,  or  fund  capability  upgrades 
over  several  years  instead  of  a  shorter  period  of  time.  According  to  range 
officials,  these  challenges  are  less  of  a  problem  for  non-MRTFB  ranges 
like  the  Redstone  Test  Center  and  other  test/laboratory  activities  funded 
through  working  capital  fund  accounts,  where  customers  pay  direct  and 
overhead  costs.26 

Service  officials  discussed  several  strategies  ora  combination  of  those 
strategies  that  the  services  could  use  to  lessen  the  impact  of  budget  cuts 


24Bob  Stump  National  Defense  Authorization  Act  for  Fiscal  Year  2003,  Pub.  L.  No.  107- 
314,  §232  (2002). 

25The  Director  of  the  Test  Resource  Management  Center  did  not  certify  the  Army’s  budget 
twice  during  this  period.  As  a  result,  the  Army  added  additional  funding  to  its  major  test 
and  evaluation  investment  account  to  address  concerns.  The  Director  of  the  Test 
Resource  Management  Center  has  not  yet  certified  any  of  the  services’  fiscal  year  2012 
proposed  test  and  evaluation  budgets. 

26Charging  customers  for  overhead  costs  does  not  reduce  DOD’s  budget  needs  for 
overhead  activities.  Rather,  it  shifts  the  burden  of  paying  for  these  costs  from  ranges  to 
customers. 
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and  funding  concerns.  One  strategy  is  that  the  ranges  may  have  to  curtail 
certain  testing,  mothball  or  close  test  facilities,  or  consolidate  test 
capabilities.  Another  option  could  be  to  move  certain  test  capabilities  out 
of  the  MRTFB  management  structure,  thereby  allowing  ranges  to  charge 
customers  the  full  cost  of  testing.  Additionally,  on  the  basis  of  our  range 
observations,  service  officials  said  more  specific  guidance  for  interpreting 
financial  management  regulations  on  what  constitutes  direct  and 
overhead  charges  for  MRTFB  operations  could  be  developed  to  clarify 
and  standardize  the  types  of  costs  that  can  be  passed  on  to  customers. 
Service  test  and  evaluation  executives  and  the  Deputy  Assistant 
Secretary  for  Developmental  Test  and  Evaluation  believe  the  ranges’ 
current  policy  interpretation  is  too  restrictive,  inconsistent  across  the 
MRTFB,  and  constrains  service  options  as  they  strive  to  sustain  MRTFB 
capabilities.  The  Test  Resource  Management  Center  recently  established 
a  team  to  study  the  issue. 


Services  Lack  Common 
Range  Performance 
Measures  to  Aid  Decision 
Making 


The  services  have  not  implemented  common  range  performance 
measures  that  would  help  them  justify  funding  and  assist  them  in  making 
workforce  decisions,  how  best  to  allocate  funding,  or  make  difficult 
decisions  about  mothballing,  closing,  or  consolidating  test  capabilities,  if 
necessary.  Although  ranges  collect  performance  data  relevant  to  their 
operations,  these  indicators  may  not  be  useful  in  making  higher-level 
infrastructure  decisions  that  cut  across  several  ranges.  According  to 
service  officials,  it  is  very  difficult  to  develop  a  common  set  of 
performance  measures  because  of  the  uniqueness  of  each  range  and  its 
variable  capacity.  We  have  found  that  performance  measures  can  assist 
managers  in  making  decisions  about  future  strategies,  planning  and 
budgeting,  identifying  priorities,  and  allocating  resources.27 


Some  efforts  are  under  way  to  provide  decision  makers  more  information, 
but  they  are  still  in  process  and  are  not  yet  approved  or  implemented.  The 
Test  Resource  Management  Center  has  sponsored  an  effort  to  develop  a 
comprehensive  set  of  range  metrics.  According  to  the  services,  early 
efforts  were  not  successful  and  were  not  well  received.  While  the  Air 
Force  plans  to  use  metrics  resulting  from  the  Test  Resource  Management 
Center’s  effort,  the  Army  is  evaluating  the  development  of  a  readiness 


27GAO,  Managing  for  Results:  Enhancing  Agency  Use  of  Performance  Information  for 
Management  Decision  Making,  GAO-05-927  (Washington,  D.C.:  Sept.  9,  2005). 
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reporting  system  and  a  workload  model  for  its  ranges  that  could  provide  a 
better  basis  for  investment  or  divestment  decisions.  The  Navy  is  also  in 
the  process  of  developing  metrics  for  its  MRTFB  test  capabilities  on  the 
systems’  condition,  capacity,  competency,  and  importance.  Once 
developed,  these  metrics  are  expected  to  assist  decision  makers  in 
directing  future  investment  funding  and  to  ensure  test  capabilities  are 
adequate  to  support  programs. 


Conclusions 


The  Reform  Act  points  to  the  need  for  DOD  to  develop  more  robust 
systems  engineering  and  developmental  test  and  evaluation  capabilities. 
DOD  established  new  organizations  and  the  services  developed  resource 
plans  in  order  to  increase  emphasis  on  these  activities.  From  an 
organizational  standpoint,  the  offices  of  the  Deputy  Assistant  Secretaries 
for  Systems  Engineering  and  Developmental  Test  and  Evaluation  are 
providing  assistance  to  program  managers  in  developing  acquisition 
plans  and  providing  oversight  of  program  efforts  for  the  Under  Secretary 
of  Defense  for  Acquisition,  Technology  and  Logistics  and  the  Congress. 
However,  the  developmental  testing  office  is  not  as  robust  or  efficient  as  it 
could  be  in  part  due  to  resource  and  organizational  constraints.  In 
addition,  while  the  military  services  have  made  significant  progress  to 
date  in  increasing  their  systems  engineering  and  developmental  testing 
workforce  capabilities,  planned  workforce  reductions  could  offset  these 
gains. 

It  is  incumbent  upon  DOD  to  provide  the  most  effective  systems 
engineering  and  developmental  testing  capability  it  can  afford.  It  should 
have  a  sound  analytical  basis  for  establishing  and  resourcing  that 
capability.  However,  it  is  not  clear  that  DOD  is  yet  at  this  point,  especially 
for  developmental  testing.  DOD  has  not  conducted  analysis  on  the  right 
size  of  the  developmental  test  and  evaluation  office  or  captured  data  that 
would  reinforce  range  funding  and  workforce  actions,  or  suggest  needed 
adjustments.  Additionally,  the  Deputy  Assistant  Secretary  for 
Developmental  Test  and  Evaluation  believes  that  there  may  also  be  a 
statutory  provision  that  limits  his  ability  to  achieve  efficiencies  and 
address  office  challenges.  To  the  extent  DOD  cannot  provide  adequate 
systems  engineering  and  developmental  testing  support  to  its  weapons 
portfolio,  the  risks  of  executing  the  portfolio  within  cost  and  schedule  are 
increased. 
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Recommendations  for 
Executive  Action 


We  recommend  that  the  Secretary  of  Defense  take  the  following  two 
actions: 

•  assess  the  resources  and  influence  needed  by  the  developmental  test 
and  evaluation  office  to  assist  and  oversee  defense  acquisition 
programs,  including 

•  the  number  of  defense  acquisition  programs  that  can  be 
supported  by  different  developmental  test  and  evaluation  office 
staffing  levels,  including  specifying  the  total  number  of  personnel, 
the  mix  of  government  and  contractor  employees,  and  the  number 
of  senior  executive  service  personnel  needed  for  each  of  these 
staffing  levels; 

•  whether  the  Test  Resource  Management  Center  and  the  office  of 
the  Deputy  Assistant  Secretary  for  Developmental  Test  and 
Evaluation  should  be  combined  or  resources  shifted  between 
organizations  to  more  effectively  support  the  activities  of  both 
organizations  and  if  so,  identify  for  Congress  any  statutory 
revisions  that  would  be  necessary;  and 

•  the  proper  reporting  channel,  taking  into  account  the  decision  on 
whether  or  not  to  combine  the  organizations,  the  statutory 
oversight  requirements,  and  the  level  of  influence  needed  to 
oversee  and  assess  program  office  developmental  testing  and 
service  budgeting  activities. 

•  develop  a  plan  to  implement  the  results  of  the  assessment. 


We  also  recommend  that  the  Secretary  of  Defense,  with  input  from  the 
military  services,  take  the  following  two  actions: 

•  develop  metrics  to  assess  the  MRTFB  test  capabilities  (expanding  to 
DOD  non-MRTFB,  and  non-DOD  government  test  facilities  once  an 
approved  set  of  metrics  are  in  place  supporting  the  MRTFB),  justify 
funding,  and  assist  in  making  decisions  on  the  right-sizing  of 
personnel,  how  best  to  allocate  funding,  or  make  future  decisions  on 
whether  to  mothball,  shut  down,  or  consolidate  test  facilities.  These 
efforts  should  be  coordinated  with  the  Test  Resource  Management 
Center. 

•  report  the  impact  budget  cuts  reflected  in  the  fiscal  year  2012  budget, 
as  well  as  the  insourcing  policy  clarification,  will  have  on  their  (1)  total 
workforce  (civilians,  military,  and  contractors)  that  support  both  of 
these  activities  and  (2)  ability  to  meet  program  office  systems 
engineering  and  developmental  test  and  evaluation  needs. 
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Matter  for 

Congressional 

Consideration 


Contingent  upon  the  results  of  DOD’s  assessment,  the  Congress  may 
want  to  consider  revising  any  applicable  statutory  provisions  necessary  to 
allow  for  DOD  to  combine  or  shift  resources  between  the  Test  Resource 
Management  Center  and  the  office  of  the  Deputy  Assistant  Secretary  for 
Developmental  Test  and  Evaluation. 


Agency  Comments 
and  Our  Evaluation 


DOD  provided  us  with  written  comments  on  a  draft  of  this  report.  DOD 
concurred  with  two  recommendations  and  partially  concurred  with  two 
others.  DOD  offered  suggested  wording  changes  to  the  recommendations 
where  it  partially  agreed  to  offer  greater  clarity.  We  agreed  with  the 
suggested  changes  and  reworded  our  recommendations  accordingly. 
DOD’s  comments  appear  in  appendix  II.  DOD  also  provided  technical 
comments,  which  we  incorporated  as  appropriate  in  the  report. 

In  its  response,  DOD  noted  that  the  Deputy  Assistant  Secretary  for 
Developmental  Test  and  Evaluation  has  directed  a  study  to  assess  the 
resources  and  influence  needed  by  the  developmental  test  and  evaluation 
office  to  assist  and  oversee  defense  acquisition  programs.  The  Deputy 
Assistant  Secretary  for  Development  Test  and  Evaluation  will  develop  a 
plan  to  implement  actionable  recommendations  from  the  results  of  the 
assessment,  with  the  approval  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics.  The  Deputy  Assistant  Secretary 
for  Developmental  Test  and  Evaluation  also  plans  to  establish  a  working 
group,  with  participation  from  the  military  services,  to  develop  metrics  to 
assess  MRTFB  test  capabilities,  justify  funding,  and  assist  in  making 
decisions  on  human  capital  and  test  facilities  management.  Finally,  the 
Deputy  Assistant  Secretaries  for  Systems  Engineering  and 
Developmental  Test  and  Evaluation  plan  to  identify  any  impacts  to  the 
state  of  the  workforce  due  to  funding  modifications  or  DOD’s  workforce 
policy  updates  in  their  joint  annual  report  to  the  Congress. 


We  are  sending  copies  of  this  report  to  the  Secretary  of  Defense,  the 
Director  of  the  Office  of  Management  and  Budget,  and  interested 
congressional  committees.  We  will  also  make  copies  available  at  no 
charge  on  the  GAO  web  site  at  http://www.gao.gov. 

If  you  have  any  questions  about  this  report  or  need  additional  information, 
please  contact  me  at  (202)  512-4841  or  sullivanm@gao.gov.  Contact 
points  for  our  Offices  of  Congressional  Relations  and  Public  Affairs  may 
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be  found  on  the  last  page  of  this  report.  Key  contributors  to  this  report  are 
listed  in  appendix  III. 


Michael  J.  Sullivan 
Director 

Acquisition  and  Sourcing  Management 
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Appendix  I:  Scope  and  Methodology 


This  report  examines  the  military  services’  systems  engineering  and 
developmental  testing  workforce  capabilities  and  the  Department  of 
Defense’s  (DOD)  efforts  to  implement  Weapon  Systems  Acquisition 
Reform  Act  of  2009  (Reform  Act)  requirements.1  Specifically,  we 
examined  (1)  the  progress  DOD  has  made  in  implementing  the  Reform 
Act’s  systems  engineering  and  developmental  testing  requirements  and 
(2)  whether  there  are  challenges  at  the  military  service  level  that  could 
affect  their  systems  engineering  and  developmental  testing  activities. 

To  measure  DOD’s  progress  in  implementing  Reform  Act  requirements 
related  to  systems  engineering  and  developmental  testing,  we 
interviewed  officials  and  collected  pertinent  documents  from  the  offices  of 
the  Deputy  Assistant  Secretary  of  Defense  for  Systems  Engineering  and 
Developmental  Test  and  Evaluation.  Specifically,  we  collected  information 
on  their  efforts  to  develop  additional  policy  and  guidance;  review  and 
approve  acquisition  planning  documents  of  major  acquisition  programs; 
monitor  and  review  activities  of  major  acquisition  programs;  and  develop 
guidance  for  the  development  of  performance  metrics  to  use  on  weapon 
acquisition  programs.  We  also  reviewed  their  staffing  plans  and 
questioned  the  Deputy  Assistant  Secretaries,  as  well  as  former  DOD 
developmental  testing  experts  on  whether  each  of  the  offices  has  the 
necessary  amount  of  resources  and  influence  to  fulfill  their  missions. 

In  order  to  determine  whether  there  are  challenges  at  the  military  service 
level  that  could  affect  future  systems  engineering  and  developmental 
testing  activities,  we  looked  at  planned  and  actual  workforce  growth  and 
developmental  test  range  activities,  and  specifically  at  the  following. 

•  We  collected  and  compared  the  military  services’  original  workforce 
growth  projections  with  their  current  workforce  projections  for  the 
systems  engineering  and  test  and  evaluation  career  fields.  For  this 
report,  when  we  use  the  term  systems  engineering  career  field,  we 
are  referring  to  the  Systems  Planning,  Research  Development,  and 
Engineering — Systems  Engineering  and  Program  Systems 
Engineering  career  fields.  The  Air  Force  and  Army’s  original  plans 


Vub.  L.  No.  111-23. 
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were  based  on  their  fiscal  year  2008  budget  estimate  submissions2 
and  the  Navy’s  was  based  on  its  fiscal  year  2009  budget  estimate 
submission.  Current  plans  for  each  of  the  services  were  based  on 
their  fiscal  year  2012  budget  estimate  submissions.  Workforce  data 
covered  the  fiscal  years  2009-2015  timeframe  for  both  the  original  and 
current  plans.  We  interviewed  officials  within  the  Office  of  the 
Secretary  of  Defense  and  military  services  to  determine  the 
underlying  causes  for  variances  between  the  two  plans  and  to 
determine  how  the  Secretary  of  Defense’s  cost  efficiency  measures 
are  affecting  new  hiring  and  insourcing  plans. 

Navy  officials  stated  that  they  used  fiscal  year  2009  as  their  baseline 
because  before  that  time,  information  from  various  commands  was 
not  centralized  and  they  could  not  verify  the  accuracy  of  the  numbers. 
Navy  officials  stated  that  they  have  confidence  in  the  validity  of  the 
numbers  that  support  the  fiscal  year  2009  baseline  because  factors 
that  affect  workforce  counts,  such  as  recodes,  decodes, 
reassignments,  and  attrition,  are  treated  consistently. 

•  We  compared  the  services’  baseline  workforce  data  with  end  of  fiscal 
year  2010  workforce  numbers  to  determine  how  much  growth 
occurred  in  the  two  career  fields  over  the  past  2  fiscal  years.  The  Air 
Force’s  and  Army’s  workforce  baseline  was  the  end  of  fiscal  year 
2008  and  the  Navy’s  was  the  end  of  fiscal  year  2009.  We  assessed 
the  reliability  of  these  data  by  interviewing  knowledgeable  officials  in 
the  Air  Force,  Army,  and  Navy  about  the  processes  they  use  to 
ensure  both  the  integrity  and  reliability  of  their  manpower  workforce 
databases  used  to  track  acquisition  personnel.  We  determined  that 
the  data  were  sufficiently  reliable  for  the  purposes  of  this  report. 

•  We  conducted  site  visits  to  12  ranges  and  facilities — 1 1  of  these  are 
designated  as  part  of  the  Major  Range  and  Test  Facility  Base 
(MRTFB)  and  one  is  a  non-MRTFB  range — to  determine  challenges 
affecting  DOD’s  ability  to  conduct  developmental  testing.  We  focused 
our  discussions  on  a  broad  range  of  topics,  including  funding, 
workforce,  range  facilities  and  instrumentation,  encroachment, 


2The  Future  Years  Defense  Program,  a  DOD  centralized  report  consisting  of  thousands  of 
program  elements,  provides  information  on  DOD’s  current  and  planned  outyear  budget 
requests  and  is  one  of  the  principal  tools  available  to  help  inform  DOD  and  the  Congress 
about  resource  data  relating  to  these  challenging  trade-offs.  The  reporting  requirement  for 
it  is  codified  at  10  U.S.C.  §  221. 
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contractor  duplication  of  DOD  test  facilities,  and  test  resource  capacity 
and  demand.  We  selected  four  ranges  per  service  based  on 
geographical  diversity,  level  of  funding,  type  of  testing,  and 
recommendations  from  the  Office  of  the  Secretary  of  Defense  and  the 
services.  The  test  ranges  we  visited  were:  Aberdeen  Test  Center, 
Maryland;  Air  Force  Flight  Test  Center,  California;  Arnold  Engineering 
Development  Center,  Tennessee;  Atlantic  Undersea  Test  and 
Evaluation  Center,  Bahamas;  46th  Test  Group,  New  Mexico;  46th 
Test  Wing,  Florida;  High  Energy  Laser  Systems  Test  Facility,  New 
Mexico;  Naval  Air  Warfare  Center  -  Aircraft  Division,  Maryland;  Naval 
Air  Warfare  Center  -  Weapons  Division,  California;  Pacific  Missile 
Range  Facility,  Hawaii;  Redstone  Test  Center,  Alabama;  and  White 
Sands  Missile  Range,  New  Mexico. 

•  Finally,  we  compared  developmental  test  range  funding  included  in 
the  President’s  Budget  Future  Year’s  Defense  Plan  for  fiscal  year 
201 1  and  2012.  We  discussed  differences  between  the  two  plans  and 
how  decreases  in  the  fiscal  year  2012  plan  would  affect 
developmental  testing  activities  with  officials  from  the  Test  Resource 
Management  Center,  the  military  services,  and  the  office  of  the 
Deputy  Assistant  Secretary  of  Defense  for  Developmental  Test  and 
Evaluation.  We  also  discussed  with  these  officials  the  progress  DOD 
and  the  services  have  made  in  developing  metrics  that  could  be  used 
to  make  workforce  and  investment  decisions. 

We  conducted  this  performance  audit  from  September  2010  to 
September  2011  in  accordance  with  generally  accepted  government 
auditing  standards.  Those  standards  require  that  we  plan  and  perform  the 
audit  to  obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable 
basis  for  our  findings  and  conclusions  based  on  our  audit  objectives.  We 
believe  that  the  evidence  obtained  provides  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives. 
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ASSISTANT  SECRETARY  OF  DEFENSE 


3030  DEFENSE  PENTAGON 
WASHINGTON.  DC  2030  1  -3030 


RESEARCH 
AND  ENGINEERING 


SEP  9  20 If 


Mr.  Michael  Sullivan 

Director.  Acquisition  and  Sourcing  Management 
U.S.  Government  Accountability  Office 
441  G  Street.  N.W, 

Washington.  DC  20548 

Dear  Mr.  Sullivan. 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  Draft  Report  1 1-806. 

W  CAPONS  AC  QUISI TION  Rf  FORM:  Actions  Needed  to  Address  Systems  engineering  and 
Developmental  Testing  Challenges."  dated  August  5.  201 1.  (GAO  Code  120936).  Detailed 
comments  on  the  report  recommendations  arc  enclosed.  Detailed  comments  on  factual 
information  within  the  body  of  the  reporl  have  been  forwarded  to  the  GAO  action  officer 
separately. 

The  Department  appreciates  the  opportunity  to  respond  to  your  dratt  report  and  looks 
forward  to  working  with  you  as  we  continue  to  ensure  a  strong  and  capable  Defense  acquisition 
capability. 


Sincerely. 


Zachary  J  I  cmnios 


enclosure: 
As  slated 
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GAO  Draft  Report  Dated  August  5.  2011 
GAO-1 1-806  (GAO  CODE  120936) 

“WEAPONS  ACQUISITION  REFORM:  ACTIONS  NEEDED  TO  ADDRESS  SYSTEMS 
ENGINEERING  AND  DEVELOPMENTAL  TESTING  CHALLENGES” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATIONS 


RECOMMENDATION  I  I  he  GAO  recommends  that  the  Secretary  of  Defense  assess  the 
resources  and  influence  needed  by  die  developmental  test  and  evaluation  office  to  assist  and 
oversee  defense  acquisition  programs,  including 

•  the  number  of  defense  acquisition  programs  that  can  be  supported  by  different 
developmental  test  and  evaluation  office  staffing  levels,  including  specifying  the  total 
number  of  personnel,  the  mix  of  government  and  contractor  employees,  and  the  number 
of  senior  executive  service  personnel  needed  for  each  of  these  staffing  levels; 

•  whether  the  I  cst  Resource  Management  Center  and  the  office  of  the  Deputy  Assistant 
Secretary  lor  Developmental  Test  and  (  valuation  should  be  combined  or  resources 
shilled  between  organizations  to  support  developmental  testing  activities  and  if  so. 
identify  for  Congress  any  statutory  revisions  that  would  be  necessary  ;  and 

•  the  proper  reporting  channel,  taking  into  account  the  decision  on  whether  or  not  to 
combine  the  organizations,  the  statutory  oversight  requirements,  and  the  level  of 
influence  needed  to  oversee  and  assess  program  office  developmental  testing  and  service 
budgeting  activities. 

DoD  RESPONSE:  Partially  Concur.  As  written,  the  language  is  overly  restrictive.  The  second 
suh-bullct  recommends  assessing  "whether  the  I  cst  Resource  Management  Center  and  the  office 
ofthe  Deputy  Assistant  Secretary  for  Developmental  lest  and  Evaluation  should  be  combined  or 
resources  shifted  between  organizations  to  support  developmental  testing  activities...”  DoD 
recommends  amending  the  language  to  “...whether  the  Test  Resource  Management  Center  and 
the  office  of  the  Deputy  Assistant  Secretary  fiir  Developmental  Test  and  Evaluation  should  be 
combined  or  resources  shifted  between  organizations  to  more  effectively  support  the  activities  of 
both  organizations. . .” 

Rationale.  I  he  Under  Secretary  ol  Delense  Ibr  Acquisition,  Technology  and  Logistics  tasked 
the  Deputy  Assistant  Secretary  of  Defense  for  Developmental  Test  and  Evaluation  to  review  the 
Ivvo  organizations  to  identify  additional  elficicncies.  develop  alternatives  and  to  report  the 
findings  to  the  Under  Secretary  of  Defense  for  Acquisition.  Technologv  and  Logistics  for  review 
and  decision. 

The  Deputy  Assistant  Secretary  of  Defense  for  Developmental  Test  and  Evaluation  has  directed 
a  study  to  assess  the  resources  and  influence  needed  by  the  developmental  test  and  evaluation 
office  to  assist  and  oversee  defense  acquisition  programs 
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Appendix  II:  Comments  from  the  Department 
of  Defense 


RECOMMENDATION  2:  The  GAO  recommends  lhal  the  Secretary  of  Defense  develop  a  plan 
to  implement  the  results  of  the  assessment. 

DoD  RESPONSE:  Concur.  With  the  Under  Secretary  of  Defense  for  Acquisition.  Technology 
iind  Logistics  approval,  the  Deputy  Assistant  Secretary  of  Defense  for  Developmental  l  est  and 
Evaluation,  serving  concurrently  as  the  Director  of  the  Test  Resource  Management  Center,  will 
develop  a  plan  to  implement  actionable  recommendations  from  the  results  of  the  assessment 


RECOMMENDATION  3:  The  GAO  recommends  that  the  Secretary  of  Defense,  with  input 
from  the  military  services,  develop  metrics  to  assess  development  test  capabilities,  justify 
funding,  and  assist  in  making  decisions  on  where  to  cut  personnel,  how  best  to  allocate  funding, 
or  make  future  decisions  on  whether  to  mothball,  shut  down,  or  consolidate  test  facilities  These 
elTorts  should  be  coordinated  with  the  Test  Resource  Management  Center. 

DoD  RESPONSE:  Partially  Concur.  As  written,  the  scope  of  the  recommendation  is  too  broad. 
This  recommendation  states,  "GAO  recommends  that  the  Secretary  of  Defense,  w  ith  input  from 
the  military  Services,  develop  metrics  to  assess  development  test  capabilities,  justify  funding 
DoD  recommends  amending  the  language  to  “. .  develop  metrics  to  assess  the  Major  Range 
and  1  est  Facility  Base  (MRTFB)  test  capabilities  (expanding  to  DoD  non-MRTFB,  and  non- 
DoD  government  test  facilities  once  an  approv  ed  set  of  metrics  are  in  place  supporting  the 
MRTEB),  justify  funding  . . .” 

Rationale:  1  he  term  "development  test  capabilities"  encompasses  a  w  ide  range  of  DoD  MRTEB. 
DoD  non-MRTFB.  and  non-DoD  government  test  facilities.  I  he  primary  focus  of  the  "Impacts 
of  Proposed  Budget  Cuts  on  Ranges"  paragraphs  on  pages  1 8-22  of  the  draft  GAO  report  is 
clearly  the  MRTFB. 

Also  as  written,  the  focus  is  about  reducing  instead  of  "right-sizing"  test  capabilities. 

Recommend  change  from  "...  assist  in  making  decisions  on  where  to  cut  personnel...''  to  "assist 
in  making  decisions  on  the  "right-sizing"  of  personnel.  " 

The  Deputy  Assistant  Secretary  of  Defense  for  Developmental  Test  and  Evaluation,  serv  ing 
concurrently  as  the  Director  of  the  Test  Resource  Management  Center,  will  establish  a  working 
group,  with  participation  from  the  military  services,  to  develop  metrics  to  assess  MRTFB  test 
capabilities,  justify  funding,  and  assist  in  making  decisions  on  human  capital  and  test  facilities 
management. 


RECOMMENDATION  4:  The  GAO  recommends  that  the  Secretary  of  Defense,  with  input 
from  the  military  services,  report  the  impact  budget  cuts  reflected  in  the  fiscal  year  201 2  budget, 
as  well  as  the  insourcing  policy  clarification,  will  have  on  their  (1 )  total  workforce  (civilians, 
military,  and  contractors)  that  support  both  of  these  activities  and  (2)  ability  to  meet  program 
office  systems  engineering  and  developmental  test  and  evaluation  needs. 
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Appendix  II:  Comments  from  the  Department 
of  Defense 


DoD  RESPONSE:  Concur.  Each  year  the  Deputy  Assistant  Secretary  ol  Defense  for 
Developmental  I  est  &  Evaluation  (D I  &E)  and  the  Deputy  Assistant  Secretary  of  Defense  for 
Systems  Engineering  (SE),  with  inputs  from  the  military  services,  submit  an  assessment  of  the 
organization  and  capabilities  of  the  Department  of  Defense  for  systems  engineering, 
development  planning,  and  developmental  test  and  evaluation,  to  include  the  workforce,  in  the 
DT&I  <uid  SE  Joint  Annual  Report  to  Congress,  as  directed  by  the  Weapon  Systems  Acquisition 
Reform  Act  of  2009.  Any  impacts  to  the  slate  of  the  workforce  due  It'  funding  modifications  or 
Department  workforce  policy  updates  will  be  reflected  in  this  report. 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and 
policies;  and  provides  analyses,  recommendations,  and  other  assistance 
to  help  Congress  make  informed  oversight,  policy,  and  funding  decisions. 
GAO’s  commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no 
cost  is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday  afternoon, 
GAO  posts  on  its  Web  site  newly  released  reports,  testimony,  and 
correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted  products, 
go  to  www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  Web  site, 

http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 
MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Ralph  Dawn,  Managing  Director,  dawnr@gao.gov,  (202)  512-4400 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125 
Washington,  DC  20548 

Public  Affairs 

Chuck  Young,  Managing  Director,  youngd@gao.gov,  (202)  512-4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 
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